Image-based keypoint generation

ABSTRACT

Operations may comprise obtaining a plurality of light detection and ranging (LIDAR) scans of a region. The operations may also comprise identifying a plurality of LIDAR poses that correspond to the plurality of LIDAR scans. In addition, the operations may comprise identifying, as a plurality of keyframes, a plurality of images of the region that are captured during capturing of the plurality of LIDAR scans. The operations may also comprise determining, based on the plurality of LIDAR poses, a plurality of camera poses that correspond to the keyframes. Further, the operations may comprise identifying a plurality of two-dimensional (2D) keypoints in the keyframes. The operations also may comprise generating one or more three-dimensional (3D) keypoints based on the plurality of 2D keypoints and the respective camera poses of the plurality of keyframes.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is a continuation of U.S. patent applicationSer. No. 16/912,549, filed on Jun. 25, 2020, which claims the benefit ofand priority to U.S. Provisional App. No. 62/866,362 filed Jun. 25,2019, the disclosures of each of which are hereby incorporated herein bythis reference in their entireties.

FIELD

The embodiments discussed herein are related to image-based keypointgeneration.

BACKGROUND

Autonomous vehicles, also known as self-driving cars, driverless cars,or robotic cars, may drive from a source location to a destinationlocation without requiring a human driver to control or navigate thevehicle. Automation of driving may be difficult for several reasons. Forexample, autonomous vehicles may use sensors to make driving decisionson the fly, or with little response time, but vehicle sensors may not beable to observe or detect some or all inputs that may be required oruseful to safely control or navigate the vehicle safely in someinstances. Vehicle sensors may be obscured by corners, rolling hills,other vehicles, etc. Vehicles sensors may not observe certain inputsearly enough to make decisions that may be necessary to operate thevehicle safely or to reach a desired destination. In addition, someinputs, such as lanes, road signs, or traffic signals, may be missing onthe road, may be obscured from view, or may not be readily visible, andtherefore may not be detectable by sensors. Furthermore, vehicle sensorsmay have difficulty detecting emergency vehicles, a stopped obstacle ina given lane of traffic, or road signs for rights of way.

SUMMARY

According to an aspect of the embodiment, operations may compriseobtaining a plurality of light detection and ranging (LIDAR) scans of aregion. The operations may also comprise identifying a plurality ofLIDAR poses in which each LIDAR pose of the plurality of LIDAR posescorresponds to a respective LIDAR scan of the plurality of LIDAR scans.In addition, the operations may comprise identifying, as a plurality ofkeyframes, a plurality of images of the region that are captured duringcapturing of the plurality of LIDAR scans. The operations may alsocomprise determining, based on the plurality of LIDAR poses, a pluralityof camera poses in which each respective camera pose of the plurality ofcamera poses corresponds to a respective keyframe of the plurality ofkeyframes. Further, the operations may comprise identifying a pluralityof two-dimensional (2D) keypoints, including identifying one or morerespective 2D keypoints in a corresponding keyframe of each of thekeyframes. Each respective 2D keypoint corresponds to a respectivefeature of the region as included in the corresponding keyframe. Theoperations also may comprise generating one or more three-dimensional(3D) keypoints based on the plurality of 2D keypoints and the respectivecamera poses of the respective keyframes of the plurality of keyframes.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one example technology area where some embodiments describedherein may be practiced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example overall system environment of an HD mapsystem interacting with multiple vehicle computing systems;

FIG. 2 illustrates an example system architecture of a vehicle computingsystem;

FIG. 3 illustrates an example of various layers of instructions in an HDmap application programming interface of a vehicle computing system;

FIG. 4 illustrates an example of system architecture of an online HD mapsystem;

FIG. 5 illustrates example components of an HD map;

FIGS. 6A-6B illustrate example geographical regions defined in an HDmap;

FIG. 7 illustrates example representations of lanes in an HD map;

FIGS. 8A-8B illustrate example lane elements and relationships betweenlane elements in an HD map;

FIGS. 9A-9B illustrate an example system configured to improve thealignment of a vehicle with respect to lanes, roads, other vehicles,landmarks, etc. using sensor data captured by one or more cameras of thevehicle;

FIG. 10 illustrates a flowchart an example method of image-basedlocalization for vehicles;

FIG. 11 is a flowchart of an example method of image-based keypointgeneration; and

FIG. 12 illustrates an example embodiment of a computing machine thatcan read instructions from a machine-readable medium and execute theinstructions in a processor or controller.

DETAILED DESCRIPTION Overview

Autonomous vehicles may use map data to discover some of the aboveinformation rather than relying on sensor data. However, conventionalmaps have several drawbacks that may make them difficult to use for anautonomous vehicle. For example, conventional maps may not provide thelevel of precision or accuracy that for navigation within a certainsafety threshold (e.g., accuracy within 30 centimeters (cm) or better).Further, global positioning systems (GPS) may provide accuracies ofapproximately 3-5 meters (m) but have large error conditions that mayresult in accuracies of over 100 m. This lack of accuracy may make itchallenging to accurately determine the location of the vehicle on a mapor to identify (e.g., using a map, even a highly precise and accurateone) a vehicle's surroundings at the level of precision and accuracydesired.

Furthermore, conventional maps may be created by survey teams that mayuse drivers with specially outfitted survey cars with high resolutionsensors that may drive around a geographic region and take measurements.Geographic region, region, area, etc. may be used synonymously withinthis disclosure and may be used to describe a portion of land that mayform part of the surface of the earth. The measurements may be providedto a team of map editors that may assemble one or more maps from themeasurements. This process may be expensive and time consuming (e.g.,taking weeks to months to create a comprehensive map). As a result, mapsassembled using such techniques may not have fresh data. For example,roads may be updated or modified on a much more frequent basis (e.g.,rate of roughly 5-10% per year) than a survey team may survey a givenregion. For example, survey cars may be expensive and limited in number,making it difficult to capture many of these updates or modifications.For example, a survey fleet may include a thousand survey cars. Due tothe large number of roads and the drivable distance in any given statein the United States, a survey fleet of a thousand cars may not coverthe same area at the same frequency of road changes to keep the map upto date on a regular basis and to facilitate safe self-driving ofautonomous vehicles. As a result, conventional techniques of maintainingmaps may be unable to provide data that is sufficiently accurate and upto date for the safe navigation of autonomous vehicles.

Embodiments of the present disclosure may perform global visuallocalization of an autonomous vehicle using keyframes and keypoints andmay create, maintain, and store high definition (HD) maps that mayinclude up-to-date information with high accuracy or precision given aninitial estimation of a pose and one or more images captured by one ormore cameras of an autonomous vehicle. The HD maps may be used by anautonomous vehicle to safely navigate to various destinations withouthuman input or with limited human input. In the present disclosurereference to “safe navigation” may refer to performance of navigationwithin a target safety threshold. For example, the target safetythreshold may be a certain number of driving hours without an accident.Such thresholds may be set by automotive manufacturers or governmentagencies. Additionally, reference to “up-to-date” information does notnecessarily mean absolutely up to date, but up to date within a targetthreshold amount of time. For example, a target threshold amount of timemay be one week or less such that a map that reflects any potentialchanges to a roadway that may have occurred within the past week may beconsidered “up-to-date”. Such target threshold amounts of time may varyanywhere from one month to 1 minute, or possibly even less.

The autonomous vehicle may be a vehicle capable of sensing itsenvironment and navigating without human input. An HD map may refer to amap that may store data with high precision and accuracy, for example,with accuracies of approximately 2-30 cm.

Some embodiments may generate HD maps that may contain spatial geometricinformation about the roads on which the autonomous vehicle may travel.Accordingly, the generated HD maps may include the information that mayallow the autonomous vehicle to navigate safely without humanintervention. Some embodiments may gather and use data from the lowerresolution sensors of the self-driving vehicle itself as it drivesaround rather than relying on data that may be collected by an expensiveand time-consuming mapping fleet process that may include a fleet ofvehicles outfitted with high resolution sensors to create HD maps. Theautonomous vehicles may have no prior map data for these routes or evenfor a given region. A region or area may refer to a defined area ofland. Some embodiments may provide location as a service (LaaS) suchthat autonomous vehicles of different manufacturers may gain access tothe most up-to-date map information collected, obtained, or created viathe aforementioned processes.

Some embodiments may generate and maintain HD maps that may be accurateand may include up-to-date road conditions for safe navigation of theautonomous vehicle. For example, the HD maps may provide the currentlocation of the autonomous vehicle relative to one or more lanes ofroads precisely enough to allow the autonomous vehicle to drive safelyin and to maneuver safety between one or more lanes of the roads.

HD maps may store a very large amount of information, and therefore maypresent challenges in the management of the information. For example, anHD map for a given region may be too large to store on a local storageof the autonomous vehicle. Some embodiments may provide a portion of anHD map to the autonomous vehicle that may allow the autonomous vehicleto determine its current location in the HD map, determine the featureson the road relative to the autonomous vehicle's position, determine ifit is safe to move the autonomous vehicle based on physical constraintsand legal constraints, etc. Examples of such physical constraints mayinclude physical obstacles, such as walls, barriers, medians, curbs,etc. and examples of legal constraints may include an allowed directionof travel for a lane, lane restrictions, speed limits, yields, stops,following distances, etc.

Some embodiments of the present disclosure may allow safe navigation foran autonomous vehicle by providing relatively low latency, for example,5-40 milliseconds or less, for providing a response to a request; highaccuracy in terms of location, for example, accuracy within 30 cm orbetter; freshness of data such that a map may be updated to reflectchanges on the road within a threshold time frame, for example, withindays, hours, minutes or seconds; and storage efficiency by reducing orminimizing the storage used by the HD Map.

In some embodiments, an autonomous vehicle may capture camera images ata higher frequency than other sensors such as a light detection andranging (LIDAR), GPS, radar, etc. As such, in some instances, the use ofcamera images to localization of vehicles may allow for faster updatingof the localization than by using data that may be obtained from othersensors. However, determining localization based on camera images alonemay be difficult or inaccurate.

According to one or more implementations of the present disclosure, theautonomous vehicle may comprise a vehicle computer system configured toobtain an initial pose for a vehicle in a region using sensor data fromone or more sensors of the vehicle For example, the vehicle computersystem may receive and use LIDAR data to obtain an initial pose for thevehicle in the region. As another example, the vehicle computer systemmay additionally or alternatively receive and use GPS data to obtain theinitial pose for the vehicle in the region.

As detailed below, the vehicle computer system may perform image-basedlocalization using the initial pose for the vehicle in the region. Theuse of the initial pose may improve the accuracy and reliability of theimage-based localization by directing the image-based localizationoperations to be performed with respect to certain features in theregion (e.g., certain objects or certain features of certain objects,etc.) that are within a certain distance of the initial pose.

The objects and advantages of the embodiments will be realized andachieved at least by the elements, features, and combinationsparticularly pointed out in the claims.

Both the foregoing general description and the following detaileddescription are given as examples and are explanatory and are notrestrictive of the invention, as claimed.

Embodiments of the present disclosure are explained with reference tothe accompanying drawings.

System Environment of HD Map System

FIG. 1 illustrates an example overall system environment of an HD mapsystem 100 that may interact with multiple vehicles, according to one ormore embodiments of the present disclosure. The HD map system 100 maycomprise an online HD map system 110 that may interact with a pluralityof vehicles 150 (e.g., vehicles 150 a-d) of the HD map system 100. Thevehicles 150 may be autonomous vehicles or non-autonomous vehicles.

The online HD map system 110 may be configured to receive sensor datathat may be captured by sensors of the vehicles 150 and combine datareceived from the vehicles 150 to generate and maintain HD maps. Theonline HD map system 110 may be configured to send HD map data to thevehicles 150 for use in driving the vehicles 150. In some embodiments,the online HD map system 110 may be implemented as a distributedcomputing system, for example, a cloud-based service that may allowclients such as a vehicle computing system 120 (e.g., vehicle computingsystems 120 a-d) to make requests for information and services. Forexample, a vehicle computing system 120 may make a request for HD mapdata for driving along a route and the online HD map system 110 mayprovide the requested HD map data to the vehicle computing system 120.

FIG. 1 and the other figures use like reference numerals to identifylike elements. A letter after a reference numeral, such as “105A,”indicates that the text refers specifically to the element having thatparticular reference numeral. A reference numeral in the text without afollowing letter, such as “105,” refers to any or all of the elements inthe figures bearing that reference numeral (e.g. “105” in the textrefers to reference numerals “105A” and/or “105N” in the figures).

The online HD map system 110 may comprise a vehicle interface module 160and an HD map store 165. The online HD map system 110 may be configuredto interact with the vehicle computing system 120 of various vehicles150 using the vehicle interface module 160. The online HD map system 110may be configured to store map information for various geographicalregions in the HD map store 165. The online HD map system 110 may beconfigured to include other modules than those illustrated in FIG. 1 ,for example, various other modules as illustrated in FIG. 4 and furtherdescribed herein.

In the present disclosure, a module may include code and routinesconfigured to enable a corresponding system (e.g., a correspondingcomputing system) to perform one or more of the operations describedtherewith. Additionally or alternatively, any given module may beimplemented using hardware including any number of processors,microprocessors (e.g., to perform or control performance of one or moreoperations), field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs) or any suitablecombination of two or more thereof. Alternatively or additionally, anygiven module may be implemented using a combination of hardware andsoftware. In the present disclosure, operations described as beingperformed by a module may include operations that the module may directa corresponding system to perform.

Further, the differentiation and separation of different modulesindicated in the present disclosure is to help with explanation ofoperations being performed and is not meant to be limiting. For example,depending on the implementation, the operations described with respectto two or more of the modules described in the present disclosure may beperformed by what may be considered as a same module. Further, theoperations of one or more of the modules may be divided among what maybe considered one or more other modules or submodules depending on theimplementation.

The online HD map system 110 may be configured to receive sensor datacollected by sensors of a plurality of vehicles 150, for example,hundreds or thousands of cars. The sensor data may include any data thatmay be obtained by sensors of the vehicles that may be related togeneration of HD maps. For example, the sensor data may include LIDARdata, captured images, etc. Additionally or alternatively, the sensordata may include information that may describe the current state of thevehicle 150, the location and motion parameters of the vehicles 150,etc.

The vehicles 150 may be configured to provide the sensor data 115 thatmay be captured while driving along various routes and to send it to theonline HD map system 110. The online HD map system 110 may be configuredto use the sensor data 115 received from the vehicles 150 to create andupdate HD maps describing the regions in which the vehicles 150 may bedriving. The online HD map system 110 may be configured to build highdefinition maps based on the collective sensor data 115 that may bereceived from the vehicles 150 and to store the HD map information inthe HD map store 165.

The online HD map system 110 may be configured to send HD map data tothe vehicles 150 at the request of the vehicles 150.

For example, in instances in which a particular vehicle 150 is scheduledto drive along a route, the particular vehicle computing system 120 ofthe particular vehicle 150 may be configured to provide informationdescribing the route being travelled to the online HD map system 110. Inresponse, the online HD map system 110 may be configured to provide HDmap data of HD maps related to the route (e.g., that represent the areathat includes the route) that may facilitate navigation and drivingalong the route by the particular vehicle 150.

In an embodiment, the online HD map system 110 may be configured to sendportions of the HD map data to the vehicles 150 in a compressed formatso that the data transmitted may consume less bandwidth. The online HDmap system 110 may be configured to receive from various vehicles 150,information describing the HD map data that may be stored at a local HDmap store (e.g., the local HD map store 275 of FIG. 2 ) of the vehicles150.

In some embodiments, the online HD map system 110 may determine that theparticular vehicle 150 may not have certain portions of the HD map datastored locally in a local HD map store of the particular vehiclecomputing system 120 of the particular vehicle 150. In these or otherembodiments, in response to such a determination, the online HD mapsystem 110 may be configured to send a particular portion of the HD mapdata to the vehicle 150.

In some embodiments, the online HD map system 110 may determine that theparticular vehicle 150 may have previously received HD map data withrespect to the same geographic area as the particular portion of the HDmap data. In these or other embodiments, the online HD map system 110may determine that the particular portion of the HD map data may be anupdated version of the previously received HD map data that was updatedby the online HD map system 110 since the particular vehicle 150 lastreceived the previous HD map data. In some embodiments, the online HDmap system 110 may send an update for that portion of the HD map datathat may be stored at the particular vehicle 150. This may allow theonline HD map system 110 to reduce or minimize the amount of HD map datathat may be communicated with the vehicle 150 and also to keep the HDmap data stored locally in the vehicle updated on a regular basis.

The vehicle 150 may include vehicle sensors 105 (e.g., vehicle sensors105 a-d), vehicle controls 130 (e.g., vehicle controls 130 a-d), and avehicle computing system 120 (e.g., vehicle computer systems 120 a-d).The vehicle sensors 105 may be configured to detect the surroundings ofthe vehicle 150. In these or other embodiments, the vehicle sensors 105may detect information describing the current state of the vehicle 150,for example, information describing the location and motion parametersof the vehicle 150.

The vehicle sensors 105 may comprise a camera, a LIDAR sensor, a globalnavigation satellite system (GNSS) receiver, for example, a GPSnavigation system, an inertial measurement unit (IMU), and others. Thevehicle sensors 105 may include one or more cameras that may captureimages of the surroundings of the vehicle. A LIDAR may survey thesurroundings of the vehicle by measuring distance to a target byilluminating that target with a laser light pulses and measuring thereflected pulses. The GPS navigation system may determine the positionof the vehicle 150 based on signals from satellites. The IMU may includean electronic device that may be configured to measure and report motiondata of the vehicle 150 such as velocity, acceleration, direction ofmovement, speed, angular rate, and so on using a combination ofaccelerometers and gyroscopes or other measuring instruments.

The vehicle controls 130 may be configured to control the physicalmovement of the vehicle 150, for example, acceleration, directionchange, starting, stopping, etc. The vehicle controls 130 may includethe machinery for controlling the accelerator, brakes, steering wheel,etc. The vehicle computing system 120 may provide control signals to thevehicle controls 130 on a regular and/or continuous basis and may causethe vehicle 150 to drive along a selected route.

The vehicle computing system 120 may be configured to perform varioustasks including processing data collected by the sensors as well as mapdata received from the online HD map system 110. The vehicle computingsystem 120 may also be configured to process data for sending to theonline HD map system 110. An example of the vehicle computing system 120is further illustrated in FIG. 2 and further described in connectionwith FIG. 2 .

The interactions between the vehicle computing systems 120 and theonline HD map system 110 may be performed via a network, for example,via the Internet. The network may be configured to enable communicationsbetween the vehicle computing systems 120 and the online HD map system110. In some embodiments, the network may be configured to utilizestandard communications technologies and/or protocols. The dataexchanged over the network may be represented using technologies and/orformats including the hypertext markup language (HTML), the extensiblemarkup language (XML), etc. In addition, all or some of links may beencrypted using conventional encryption technologies such as securesockets layer (SSL), transport layer security (TLS), virtual privatenetworks (VPNs), Internet Protocol security (IPsec), etc. In someembodiments, the entities may use custom and/or dedicated datacommunications technologies.

Vehicle Computing System

FIG. 2 illustrates an example system architecture of the vehiclecomputing system 120. The vehicle computing system 120 may include aperception module 210, a prediction module 215, a planning module 220, acontrol module 225, a local HD map store 275, an HD map system interface280, and an HD map application programming interface (API) 205. Thevarious modules of the vehicle computing system 120 may be configured toprocess various types of data including sensor data 230, a behaviormodel 235, routes 240, and physical constraints 245. In someembodiments, the vehicle computing system 120 may contain more or fewermodules. The functionality described as being implemented by aparticular module may be implemented by other modules.

With reference to FIG. 2 and FIG. 1 , in some embodiments, the vehiclecomputing system 120 may include a perception module 210. The perceptionmodule 210 may be configured to receive sensor data 230 from the vehiclesensors 105 of the vehicles 150. The sensor data 230 may include datacollected by cameras of the car, LIDAR, IMU, GPS navigation system,radar devices, etc. The perception module 210 may also be configured touse the sensor data 230 to determine what objects are around thecorresponding vehicle 150, the details of the road on which thecorresponding vehicle 150 is travelling, etc. In addition, theperception module 210 may be configured to process the sensor data 230to populate data structures storing the sensor data 230 and to providethe information or instructions to a prediction module 215 of thevehicle computing system 120.

The prediction module 215 may be configured to interpret the dataprovided by the perception module 210 using behavior models of theobjects perceived to determine whether an object may be moving or likelyto move. For example, the prediction module 215 may determine thatobjects representing road signs may not be likely to move, whereasobjects identified as vehicles, people, etc., may either be in motion orlikely to move. The prediction module 215 may also be configured to usebehavior models 235 of various types of objects to determine whetherthey may be likely to move. In addition, the prediction module 215 mayalso be configured to provide the predictions of various objects to aplanning module 200 of the vehicle computing system 120 to plan thesubsequent actions that the corresponding vehicle 150 may take next.

The planning module 200 may be configured to receive informationdescribing the surroundings of the corresponding vehicle 150 from theprediction module 215 and a route 240 that may indicate a destination ofthe vehicle 150 and that may indicate the path that the vehicle 150 maytake to get to the destination.

The planning module 200 may also be configured to use the informationfrom the prediction module 215 and the route 240 to plan a sequence ofactions that the vehicle 150 may to take within a short time interval,for example, within the next few seconds. In some embodiments, theplanning module 200 may be configured to specify a sequence of actionsas one or more points representing nearby locations that thecorresponding vehicle 150 may drive through next. The planning module200 may be configured to provide, to the control module 225, the detailsof a plan comprising the sequence of actions to be taken by thecorresponding vehicle 150. The plan may indicate the subsequent actionor actions of the corresponding vehicle 150, for example, whether thecorresponding vehicle 150 may perform a lane change, a turn, anacceleration by increasing the speed or slowing down, etc.

The control module 225 may be configured to determine the controlsignals that may be sent to the vehicle controls 130 of thecorresponding vehicle 150 based on the plan that may be received fromthe planning module 200. For example, if the corresponding vehicle 150is currently at point A and the plan specifies that the correspondingvehicle 150 should next proceed to a nearby point B, the control module225 may determine the control signals for the vehicle controls 130 thatmay cause the corresponding vehicle 150 to go from point A to point B ina safe and smooth way, for example, without taking any sharp turns or azig zag path from point A to point B. The path that may be taken by thecorresponding vehicle 150 to go from point A to point B may depend onthe current speed and direction of the corresponding vehicle 150 as wellas the location of point B with respect to point A. For example, if thecurrent speed of the corresponding vehicle 150 is high, thecorresponding vehicle 150 may take a wider turn compared to anothervehicle driving slowly.

The control module 225 may also be configured to receive physicalconstraints 245 as input. The physical constraints 245 may include thephysical capabilities of the corresponding vehicle 150. For example, thecorresponding vehicle 150 having a particular make and model may be ableto safely make certain types of vehicle movements such as accelerationand turns that another vehicle with a different make and model may notbe able to make safely. In addition, the control module 225 may beconfigured to incorporate the physical constraints 245 in determiningthe control signals for the vehicle controls 130 of the correspondingvehicle 150. In addition, the control module 225 may be configured tosend control signals to the vehicle controls 130 that may cause thecorresponding vehicle 150 to execute the specified sequence of actionsand may cause the corresponding vehicle 150 to move according to apredetermined set of actions. In some embodiments, the aforementionedsteps may be constantly repeated every few seconds and may cause thecorresponding vehicle 150 to drive safely along the route that may havebeen planned for the corresponding vehicle 150.

The various modules of the vehicle computing system 120 including theperception module 210, prediction module 215, and planning module 220may be configured to receive map information to perform their respectivecomputations. The corresponding vehicle 150 may store the HD map data inthe local HD map store 275. The modules of the vehicle computing system120 may interact with the map data using an HD map API 205.

The HD map API 205 may provide one or more application programminginterfaces (APIs) that can be invoked by a module for accessing the mapinformation. The HD map system interface 280 may be configured to allowthe vehicle computing system 120 to interact with the online HD mapsystem 110 via a network (not illustrated in the Figures). The local HDmap store 275 may store map data in a format that may be specified bythe online HD map system 110. The HD map API 205 may be configured toprocess the map data format as provided by the online HD map system 110.The HD map API 205 may be configured to provide the vehicle computingsystem 120 with an interface for interacting with the HD map data. TheHD map API 205 may include several APIs including a localization API250, a landmark map API 255, a 3D map API 265, a route API 270, a mapupdate API 285, etc.

The localization API 250 may be configured to determine the currentlocation of the corresponding vehicle 150, for example, where thecorresponding vehicle 150 is with respect to a given route. Thelocalization API 250 may be configured to include a localized API thatdetermines a location of the corresponding vehicle 150 within an HD mapand within a particular degree of accuracy. The vehicle computing system120 may also be configured to use the location as an accurate (e.g.,within a certain level of accuracy) relative position for making otherqueries, for example, feature queries, navigable space queries, andoccupancy map queries further described herein.

The localization API 250 may be configured to receive inputs comprisingone or more of, location provided by GPS, vehicle motion data providedby IMU, LIDAR scanner data, camera images, etc. The localization API 250may be configured to return an accurate location of the correspondingvehicle 150 as latitude and longitude coordinates. The coordinates thatmay be returned by the localization API 250 may be more accuratecompared to the GPS coordinates used as input, for example, the outputof the localization API 250 may have precision ranging within from 2-30cm. In some embodiments, the vehicle computing system 120 may beconfigured to invoke the localization API 250 to determine the locationof the corresponding vehicle 150 periodically based on the LIDAR usingscanner data, for example, at a frequency of 10 Hertz (Hz).

The vehicle computing system 120 may also be configured to invoke thelocalization API 250 to determine the vehicle location at a higher rate(e.g., 60 Hz) if GPS or IMU data is available at that rate. In addition,vehicle computing system 120 may be configured to store as internalstate, location history records to improve accuracy of subsequentlocalization calls. The location history record may store history oflocation from the point-in-time, when the corresponding vehicle 150 wasturned off/stopped, etc. The localization API 250 may include alocalize-route API that may be configured to generate an accurate (e.g.,within a specified degree of accuracy) route specifying lanes based onthe HD maps. The localize-route API may be configured to receive asinput a route from a source to a destination via one or more third-partymaps and may be configured to generate a high precision (e.g., within aspecified degree of precision such as within 30 cm) route represented asa connected graph of navigable lanes along the input routes based on HDmaps.

The landmark map API 255 may be configured to provide a geometric andsemantic description of the world around the corresponding vehicle 150,for example, description of various portions of lanes that thecorresponding vehicle 150 is currently travelling on. The landmark mapAPIs 255 comprise APIs that may be configured to allow queries based onlandmark maps, for example, fetch-lanes API and fetch-features API. Thefetch-lanes API may be configured to provide lane information relativeto the corresponding vehicle 150 and the fetch-features API. Thefetch-lanes API may also be configured to receive, as input, a location,for example, the location of the corresponding vehicle 150 specifiedusing latitude and longitude and return lane information relative to theinput location. In addition, the fetch-lanes API may be configured tospecify a distance parameter indicating the distance relative to theinput location for which the lane information may be retrieved. Further,the fetch-features API may be configured to receive informationidentifying one or more lane elements and to return landmark featuresrelative to the specified lane elements. The landmark features mayinclude, for each landmark, a spatial description that may be specificto the type of landmark.

The 3D map API 265 may be configured to provide access to the spatial3-dimensional (3D) representation of the road and various physicalobjects around the road as stored in the local HD map store 275. The 3Dmap APIs 265 may include a fetch-navigable-surfaces API and afetch-occupancy-grid API. The fetch-navigable-surfaces API may beconfigured to receive as input identifiers for one or more lane elementsand return navigable boundaries for the specified lane elements. Thefetch-occupancy-grid API may also be configured to receive a location asinput, for example, a latitude and a longitude of the correspondingvehicle 150, and return information describing occupancy for the surfaceof the road and all objects available in the HD map near the location.The information describing occupancy may include a hierarchicalvolumetric grid of some or all positions considered occupied in the HDmap. The occupancy grid may include information at a high resolutionnear the navigable areas, for example, at curbs and bumps, andrelatively low resolution in less significant areas, for example, treesand walls beyond a curb. In addition, the fetch-occupancy-grid API maybe configured to detect obstacles and to change direction, if necessary.

The 3D map APIs 265 may also include map-update APIs, for example,download-map-updates API and upload-map-updates API. Thedownload-map-updates API may be configured to receive as input a plannedroute identifier and download map updates for data relevant to allplanned routes or for a specific planned route. The upload-map-updatesAPI may be configured to upload data collected by the vehicle computingsystem 120 to the online HD map system 110. The upload-map-updates APImay allow the online HD map system 110 to keep the HD map data stored inthe online HD map system 110 up to date based on changes in map datathat may be observed by sensors of vehicles 150 driving along variousroutes.

The route API 270 may be configured to return route informationincluding a full route between a source and destination and portions ofa route as the corresponding vehicle 150 travels along the route. The 3Dmap API 265 may be configured to allow querying of the online HD mapsystem 110. The route APIs 270 may include an add-planned-routes API anda get-planned-route API. The add-planned-routes API may be configured toprovide information describing planned routes to the online HD mapsystem 110 so that information describing relevant HD maps may bedownloaded by the vehicle computing system 120 and kept up to date. Theadd-planned-routes API may be configured to receive as input, a routespecified using polylines expressed in terms of latitudes and longitudesand also a time-to-live (TTL) parameter specifying a time period afterwhich the route data may be deleted. Accordingly, the add-planned-routesAPI may be configured to allow the vehicle 150 to indicate the route thevehicle 150 is planning on taking in the near future as an autonomoustrip. The add-planned-route API may also be configured to align theroute to the HD map, record the route and its TTL value, and determinethat the HD map data for the route stored in the vehicle computingsystem 120 is up to date. The get-planned-routes API may be configuredto return a list of planned routes and to provide information describinga route identified by a route identifier.

The map update API 285 may be configured to manage operations related toupdating of map data, both for the local HD map store 275 and for the HDmap store 165 stored in the online HD map system 110. Accordingly,modules in the vehicle computing system 120 may be configured to invokethe map update API 285 for downloading data from the online HD mapsystem 110 to the vehicle computing system 120 for storing in the localHD map store 275. The map update API 285 may also be configured to allowthe vehicle computing system 120 to determine whether the informationmonitored by the vehicle sensors 105 indicates a discrepancy in the mapinformation provided by the online HD map system 110 and upload data tothe online HD map system 110 that may result in the online HD map system110 updating the map data stored in the HD map store 165 that isprovided to other vehicles 150.

FIG. 3 illustrates an example of various layers of instructions in theHD map API 205 of the vehicle computing system 120. Differentmanufacturers of vehicles may have different procedures or instructionsfor receiving information from vehicle sensors 105 and for controllingthe vehicle controls 130. Furthermore, different vendors may providedifferent computer platforms with autonomous driving capabilities, forexample, collection and analysis of vehicle sensor data. Examples of acomputer platform for autonomous vehicles include platforms providedvendors, such as NVIDIA, QUALCOMM, and INTEL. These platforms mayprovide functionality for use by autonomous vehicle manufacturers in themanufacture of vehicles 150. A vehicle manufacturer may use any one orseveral computer platforms for vehicles 150.

The online HD map system 110 may be configured to provide a library forprocessing HD maps based on instructions specific to the manufacturer ofthe vehicle and instructions specific to a vendor specific platform ofthe vehicle. The library may provide access to the HD map data and mayallow the vehicle 150 to interact with the online HD map system 110.

As illustrated in FIG. 3 , the HD map API 205 may be configured to beimplemented as a library that includes a vehicle manufacturer adapter310, a computer platform adapter 320, and a common HD map API layer 330.The common HD map API layer 330 may be configured to include genericinstructions that may be used across a plurality of vehicle computerplatforms and vehicle manufacturers. The computer platform adapter 320may be configured to include instructions that may be specific to eachcomputer platform. For example, the common HD map API layer 330 may beconfigured to invoke the computer platform adapter 320 to receive datafrom sensors supported by a specific computer platform. The vehiclemanufacturer adapter 310 may be configured to include instructionsspecific to a vehicle manufacturer. For example, the common HD map APIlayer 330 may be configured to invoke functionality provided by thevehicle manufacturer adapter 310 to send specific control instructionsto the vehicle controls 130.

The online HD map system 110 may be configured to store computerplatform adapters 320 for a plurality of computer platforms and vehiclemanufacturer adapters 310 for a plurality of vehicle manufacturers. Theonline HD map system 110 may be configured to determine the particularvehicle manufacturer and the particular computer platform for a specificvehicle 150. The online HD map system 110 may also be configured toselect the vehicle manufacturer adapter 310 for the particular vehiclemanufacturer and the computer platform adapter 320 the particularcomputer platform of a specific vehicle. In addition, the online HD mapsystem 110 may be configured to send instructions of the selectedvehicle manufacturer adapter 310 and the selected computer platformadapter 320 to the vehicle computing system 120 of that specificvehicle. The vehicle computing system 120 of that specific vehicle maybe configured to install the received vehicle manufacturer adapter 310and the computer platform adapter 320. The vehicle computing system 120may also be configured to periodically verify whether the online HD mapsystem 110 has an update to the installed vehicle manufacturer adapter310 and the computer platform adapter 320. In addition, if a more recentupdate is available compared to the version installed on the vehicle150, the vehicle computing system 120 may be configured to request andreceive the latest update and to install it.

HD Map System Architecture

FIG. 4 illustrates an example system architecture of the online HD mapsystem 110. The online HD map system 110 may be configured to include amap creation module 410, a map update module 420, a map data encodingmodule 430, a load balancing module 440, a map accuracy managementmodule 450, the vehicle interface module 160, and the HD map store 165.Some embodiments of online HD map system 110 may be configured toinclude more or fewer modules than shown in FIG. 4 . Functionalityindicated as being performed by a particular module may be implementedby other modules. In some embodiments, the online HD map system 110 maybe configured to be a distributed system comprising a plurality ofprocessing systems.

The map creation module 410 may be configured to create HD map data ofHD maps from the sensor data collected from several vehicles 150 thatare driving along various routes. The map update module 420 may beconfigured to update previously computed HD map data by receiving morerecent information (e.g., sensor data) from vehicles 150 that recentlytravelled along routes on which map information changed. For example,certain road signs may have changed or lane information may have changedas a result of construction in a region, and the map update module 420may be configured to update the HD maps and corresponding HD map dataaccordingly. The map data encoding module 430 may be configured toencode the HD map data to be able to store the data efficiently (e.g.,compress the HD map data) as well as send the HD map data to vehicles150. The load balancing module 440 may be configured to balance loadsacross vehicles 150 such that requests to receive data from vehicles 150are distributed across various vehicles 150 in a relatively uniformmanner (e.g., the load distribution between different vehicles 150 iswithin a threshold amount of each other). The map accuracy managementmodule 450 may be configured to maintain relatively high accuracy of theHD map data using various techniques even though the informationreceived from individual vehicles may not have the same degree ofaccuracy.

FIG. 5 illustrates example components of an HD map 510. The HD map 510may include HD map data of maps of several geographical regions. In thepresent disclosure, reference to a map or an HD map, such as HD map 510,may include reference to the map data that corresponds to such map.Further, reference to information of a respective map may also includereference to the map data of that map.

In some embodiments, the HD map 510 of a geographical region may includea landmark map (LMap) 520 and an occupancy map (OMap) 530. The landmarkmap 520 may comprise information describing lanes including spatiallocation of lanes and semantic information about each lane. The spatiallocation of a lane may comprise the geometric location in latitude,longitude, and elevation at high prevision, for example, precisionwithin 30 cm or better. The semantic information of a lane comprisesrestrictions such as direction, speed, type of lane (for example, a lanefor going straight, a left turn lane, a right turn lane, an exit lane,and the like), restriction on crossing to the left, connectivity toother lanes, etc.

In these or other embodiments, the landmark map 520 may compriseinformation describing stop lines, yield lines, spatial location ofcross walks, safely navigable space, spatial location of speed bumps,curb, road signs comprising spatial location, type of all signage thatis relevant to driving restrictions, etc. Examples of road signsdescribed in an HD map 510 may include stop signs, traffic lights, speedlimits, one-way, do-not-enter, yield (vehicle, pedestrian, animal), etc.

In some embodiments, the occupancy map 530 may comprise a spatial3-dimensional (3D) representation of the road and physical objectsaround the road. The data stored in an occupancy map 530 may also bereferred to herein as occupancy grid data. The 3D representation may beassociated with a confidence score indicative of a likelihood of theobject existing at the location. The occupancy map 530 may berepresented in a number of other ways. In some embodiments, theoccupancy map 530 may be represented as a 3D mesh geometry (collectionof triangles) which may cover the surfaces. In some embodiments, theoccupancy map 530 may be represented as a collection of 3D points whichmay cover the surfaces. In some embodiments, the occupancy map 530 maybe represented using a 3D volumetric grid of cells at 5-10 cmresolution. Each cell may indicate whether or not a surface exists atthat cell, and if the surface exists, a direction along which thesurface may be oriented.

The occupancy map 530 may take a large amount of storage space comparedto a landmark map 520. For example, data of 1 GB/Mile may be used by anoccupancy map 530, resulting in the map of the United States (including4 million miles of road) occupying 4×10¹⁵ bytes or 4 petabytes.Therefore, the online HD map system 110 and the vehicle computing system120 may use data compression techniques to be able to store and transfermap data thereby reducing storage and transmission costs. Accordingly,the techniques disclosed herein may help improve the self-driving ofvehicles by improving the efficiency of data storage and transmissionwith respect to self-driving operations and capabilities.

In some embodiments, the HD map 510 does may not use or rely on datathat may typically be included in maps, such as addresses, road names,ability to geo-code an address, and ability to compute routes betweenplace names or addresses. The vehicle computing system 120 or the onlineHD map system 110 may access other map systems, for example, GOOGLEMAPS, to obtain this information. Accordingly, a vehicle computingsystem 120 or the online HD map system 110 may receive navigationinstructions from a tool such as GOOGLE MAPS into a route and mayconvert the information to a route based on the HD map 510 or mayconvert the information such that it may be compatible for us on the HDmap 510.

Geographical Regions in HD Maps

The online HD map system 110 may divide a large physical area intogeographical regions and may store a representation of each geographicalregion. Each geographical region may represent a contiguous area boundedby a geometric shape, for example, a rectangle or square. In someembodiments, the online HD map system 110 may divide a physical areainto geographical regions of similar size independent of the amount ofdata needed to store the representation of each geographical region. Insome embodiments, the online HD map system 110 may divide a physicalarea into geographical regions of different sizes, where the size ofeach geographical region may be determined based on the amount ofinformation needed for representing the geographical region. Forexample, a geographical region representing a densely populated areawith a large number of streets may represent a smaller physical areacompared to a geographical region representing sparsely populated areawith very few streets. In some embodiments, the online HD map system 110may determine the size of a geographical region based on an estimate ofan amount of information that may be used to store the various elementsof the physical area relevant for the HD map.

In some embodiments, the online HD map system 110 may represent ageographic region using an object or a data record that may includevarious attributes including: a unique identifier for the geographicalregion; a unique name for the geographical region; a description of theboundary of the geographical region, for example, using a bounding boxof latitude and longitude coordinates; and a collection of landmarkfeatures and occupancy grid data.

FIGS. 6A-6B illustrate example geographical regions 610 a and 610 b thatmay be defined in an HD map according to one or more embodiments. FIG.6A illustrates a geographical region 610 a. FIG. 6B illustrates twoneighboring geographical regions 610 a and 610 b. The online HD mapsystem 110 may store data in a representation of a geographical regionthat may allow for transitions from one geographical region to anotheras a vehicle 150 drives across geographical region boundaries.

In some embodiments, as illustrated in FIG. 6 , each geographic regionmay include a buffer of a predetermined width around it. The buffer maycomprise redundant map data around one or more sides e of a geographicregion. In these or other embodiments, the buffer may be around everyside of a particular geographic region. Therefore, in some embodiments,where the geographic region may be a certain shape, the geographicregion may be bounded by a buffer that may be a larger version of thatshape. By way of example, FIG. 6A illustrates a boundary 620 for abuffer of approximately 50 m around the geographic region 610 a and aboundary 630 for a buffer of approximately 100 m around the geographicregion 610 a.

In some embodiments, the vehicle computing system 120 may switch thecurrent geographical region of the corresponding vehicle 150 from onegeographical region to a neighboring geographical region when thecorresponding vehicle 150 crosses a predetermined threshold distancewithin the buffer. For example, as shown in FIG. 6B, the correspondingvehicle 150 may start at location 650 a in the geographical region 610a. The corresponding vehicle 150 may traverse along a route to reach alocation 650 b where it may cross the boundary of the geographicalregion 610 but may stay within the boundary 620 of the buffer.Accordingly, the vehicle computing system 120 of the correspondingvehicle 150 may continue to use the geographical region 610 a as thecurrent geographical region of the vehicle. Once the correspondingvehicle 150 crosses the boundary 620 of the buffer at location 650 c,the vehicle computing system 120 may switch the current geographicalregion of the corresponding vehicle 150 to geographical region 610 bfrom geographical region 610 a. The use of a buffer may reduce orprevent rapid switching of the current geographical region of a vehicle150 as a result of the vehicle 150 travelling along a route that mayclosely track a boundary of a geographical region.

Lane Representations in HD Maps

The HD map system 100 may represent lane information of streets in HDmaps. Although the embodiments described may refer to streets, thetechniques may be applicable to highways, alleys, avenues, boulevards,paths, etc., on which vehicles 150 may travel. The HD map system 100 mayuse lanes as a reference frame for purposes of routing and forlocalization of the vehicle 150. The lanes represented by the HD mapsystem 100 may include lanes that are explicitly marked, for example,white and yellow striped lanes, lanes that may be implicit, for example,on a country road with no lines or curbs but may nevertheless have twodirections of travel, and implicit paths that may act as lanes, forexample, the path that a turning car may make when entering a lane fromanother lane.

The HD map system 100 may also store information relative to lanes, forexample, landmark features such as road signs and traffic lightsrelative to the lanes, occupancy grids relative to the lanes forobstacle detection, and navigable spaces relative to the lanes so thevehicle 150 may plan/react in emergencies when the vehicle 150 makes anunplanned move out of the lane. Accordingly, the HD map system 100 maystore a representation of a network of lanes to allow the vehicle 150 toplan a legal path between a source and a destination and to add a frameof reference for real-time sensing and control of the vehicle 150. TheHD map system 100 stores information and provides APIs that may allow avehicle 150 to determine the lane that the vehicle 150 is currently in,the precise location of the vehicle 150 relative to the lane geometry,and other relevant features/data relative to the lane and adjoining andconnected lanes.

FIG. 7 illustrates example lane representations in an HD map. FIG. 7illustrates a vehicle 710 at a traffic intersection. The HD map system100 provides the vehicle 710 with access to the map data that may berelevant for autonomous driving of the vehicle 710. This may include,for example, features 720 a and 720 b that may be associated with thelane but may not be the closest features to the vehicle 710. Therefore,the HD map system 100 may store a lane-centric representation of datathat may represent the relationship of the lane to the feature so thatthe vehicle 710 can efficiently extract the features given a lane.

The HD map data may represent portions of the lanes as lane elements.The lane elements may specify the boundaries of the lane and variousconstraints including the legal direction in which a vehicle may travelwithin the lane element, the speed with which the vehicle may drivewithin the lane element, whether the lane element may be for left turnonly, or right turn only, etc. In some embodiments, the HD map data mayrepresent a lane element as a continuous geometric portion of a singlevehicle lane. The HD map system 100 may store objects or data structuresthat may represents lane elements that may comprise informationrepresenting geometric boundaries of the lanes; driving direction alongthe lane; vehicle restriction for driving in the lane, for example,speed limit, relationships with connecting lanes including incoming andoutgoing lanes; a termination restriction, for example, whether the laneends at a stop line, a yield sign, or a speed bump; and relationshipswith road features that are relevant for autonomous driving, forexample, traffic light locations, road sign locations, etc. as part ofthe HD map data.

Examples of lane elements represented by the HD map data may include, apiece of a right lane on a freeway, a piece of a lane on a road, a leftturn lane, the turn from a left turn lane into another lane, a mergelane from an on-ramp an exit lane on an off-ramp, and a driveway. The HDmap data may represent a one-lane road using two lane elements, one foreach direction. The HD map system 100 may represents median turn lanesthat may be shared similar to a one-lane road.

FIGS. 8A-B illustrate lane elements and relations between lane elementsin an HD map. FIG. 8A illustrates an example of a T-junction in a roadillustrating a lane element 810 a that may be connected to lane element810 c via a turn lane 810 b and may be connected to lane 810 e via aturn lane 810 d. FIG. 8B illustrates an example of a Y-junction in aroad illustrating label 810 f that may be connected to lane 810 hdirectly and may be connected to lane 810 i via lane 810 g. The HD mapsystem 100 may determine a route from a source location to a destinationlocation as a sequence of connected lane elements that may be traversedto reach from the source location to the destination location.

Image Keypoint Generation

FIG. 9A illustrates an example system 900 configured to generate imagekeypoints, according to one or more embodiments of the presentdisclosure. The system 900 may also be configured to use the image-basedmap annotations to perform image-based localization as discussed indetail below. The system 900 may be analogous to the HD map system 100of FIG. 1 . The system 900 may include an online HD map system 910,vehicle interface module 960, and a HD map store 965, which may beanalogous to the online HD map system 110, vehicle interface module 160,and a HD map store 165, respectively of FIG. 1 .

The system 900 may also include the vehicle 950, which may be analogousto one of the vehicles 150 of FIG. 1 . The vehicle 950 may also includevehicle sensors 905, a vehicle computing system 920 (“computing system920”), and vehicle controls 930, which may be analogous to the vehiclesensors 105, vehicle computing system 120, and vehicle controls 130,respectively, of FIG. 1 . In the present disclosure, reference to thesystem 900 performing operations may include any applicable element ofthe system 900 performing such operations.

The system 900 (e.g., via the online HD map system 910 and/or thevehicle computing system 920) may be configured to generate one or morethree-dimensional image keypoints (“3D keypoints”). The 3D keypoints maycorrespond to features of points of interest (e.g., certain objects) ina region and may be derived from images of the region. As detailedbelow, the system 900 may be configured to use LIDAR scans to generatethe 3D keypoints and identify their respective locations in the region.The use of the LIDAR scans to perform such operations may improve theaccuracy of the 3D keypoints and their associated locations.

In some embodiments, the system 900 may be configured to annotate mapdata of an HD map 912 with keypoint data that corresponds to the 3Dkeypoints. The HD map 912 and corresponding HD map data may be stored inthe HD map store 965. Additionally, the HD map 912 and corresponding HDmap data may be communicated to the vehicle 950 via the online HD mapsystem 910 such that the HD map 912 and the corresponding HD map datamay be stored locally in data storage or memory of the computing system920. The HD map 912 and corresponding HD map data stored locally in thecomputing system 920 may be a subset of the HD map 912 and correspondingHD map data stored in the HD map store 965.

In some embodiments, the system 900 may be configured to annotate the HDmap 912 with data that identifies locations of respective areas withinthe HD map 912 that may correspond to 3D keypoints. FIG. 9B illustratesan example of a region 901 represented by the HD map 912 that mayinclude such annotations.

For example, the HD map 912 may include HD map data that identifies thelocation of each of one or more 3D keypoints for the region 901, forexample 3D keypoints 970 a-970 f (referred to collectively orgenerically as “keypoints 970”). In some instances, the keypoints 970may each respectively correspond to a respective feature in the region901 that may be of interest with respect to visual localization. Thesystem 900 may be configured to generate the keypoints 970 as followsbelow.

In some embodiments, the system 900 may obtain a plurality of LIDARscans of a region. The LIDAR scans may be obtained by one or morevehicles that have travelled in the region 901 and that have used LIDARto scan the region 901 while travelling therewithin. In someembodiments, the LIDAR scans may be obtained in real-time while thecorresponding vehicle is traveling. Additionally or alternatively, theLIDAR scans may be obtained from map data of the HD map 912 that hasbeen generated based on the LIDAR scans.

The system 900 may be configured to identify identify a plurality ofLIDAR poses in which each LIDAR pose corresponds to a respective LIDARscan of the plurality of LIDAR scans. The LIDAR poses may may refer toan alignment, orientation, location, etc. of a LIDAR scanner and/or thevehicle on which the LIDAR scanner may be disposed when scanning anarea. In instances in which the LIDAR scans are obtained from the HD mapdata, the corresponding LIDAR poses may also be stored with the HD mapdata and may be obtained therefrom. Additionally or alternatively,localization may be performed (e.g., using the LIDAR scans and map dataof the HD map) to determine the LIDAR poses. In these or otherembodiments, the LIDAR poses may be given with respect to a global frameof reference of the HD map 912. Additionally or alternatively, the LIDARposes may be obtained by comparing different LIDAR scans and using anidentified trajectory of the vehicle (e.g., the vehicle 950). In theseor other embodiments, the LIDAR poses may be with respect to a relativeframe of reference with respect to the corresponding LIDAR scans.

The system 900 may be configured to identify images that may correspondto the LIDAR scans. For example, while driving through the region 901and obtaining LIDAR scans, the vehicle that is obtaining the LIDAR scansmay also capture images of the region 901 using one or more cameras.These captured images may thus correspond to the LIDAR scans. Theidentified images may be referred to as “keyframes.”

In some embodiments, the system 900 may select as keyframes cameraimages captured with respect to camera poses that are at a givendistance interval apart from each other. The camera poses may indicatean alignment, orientation, location, etc. of a camera and/or the vehicleon which the camera may be disposed when capturing the correspondingimage. Reference to a distance between images or keyframes in thepresent disclosure may refer to distances between camera poses thatcorrespond to the images or keyframes. In some embodiments, one or moreof the images may be closer to each other than a determined minimumdistance and the system 900 may filter and/or remove the images fromselection as keyframes such that the distance between the keyframes maybe greater than the threshold distance.

In some embodiments, the system 900 may determine and/or identify twodimensional (2D) keypoints within the keyframes. For example, the system900 may be configured to identify one or more objects in the keyframesand extract one or more points of interest from the one or more objects.

For example, point of interests may be identified based on pixels orsets of pixels within an image that may differ in their intensity bymore than a predetermined value from other pixels or other sets ofpixels of the image that may be adjacent to or near the pixel or set ofpixels. Additionally or alternatively, the points of interest may beidentified as features that may be distinguishable from other portionsof the image due to differing color, size, and/or shape; areas wheretransitions may occur between pixels, features, and/or textures;features and/or objects with particular pattern or texture; points,areas, regions, etc. of high intensity within the image; points, areas,or regions with semantics within the image, etc. In some embodiments,the system 900 may identify as 2D keypoints the points of interest suchthat the 2D keypoints may correspond to features that correspond to thepoints of interest.

In some embodiments, the system 900 may identify and/or storeinformation describing the 2D keypoints based on the points of interestincluding, but not limited to features, descriptors, etc. For example,in some embodiments, the system 900 (e.g., via the computing system ofthe online HD map system 910) may identify an object in a keypframe andmay extract points of interest from or on the object. In these or otherembodiments, the system 900 may be configured to identify features ofthe points of interest and to provide a feature description of theobject. In some embodiments, the feature description may be extractedfrom a training image and may be used to identify the object whenattempting to locate the object in a new image that may contain variousother objects. As indicated above, in some embodiments, system mayidentify the 2D keypoints from or on high-contrast areas of the image,such as object edges.

In some embodiments, the system 900 may select a subset of the 2Dkeypoints based on 2D keypoints that are included in multiple of thekeyframes. For example, the system 900 may identify multiple 2Dkeypoints in multiple keyframes that may each be associated with thesame feature in the region 901. In some embodiments, the same featuremay be captured in images from various angles. For example, features maybe included in images captured by different cameras from various angles,from various lanes, from various directions, at various heights, etc. Asa result, there may be multiple keyframes that may display the samefeatures and thus may include the same 2D keypoints. In someembodiments, the system 900 may identify the subset of 2D keypointsbased on inclusion of the same 2D keypoints in multiple keyframes.

For example, the system 900 may match 2D keypoints between neighboringkeyframes. In some embodiments, the matching may be based on identifyingcommon features between multiple keyframes in which the common featurescorrespond to respective 2D keypoints of the respective keyframes. Inthese or other embodiments, the system 900 may determine match valuesbetween 2D keypoints and may identify as matching 2D keypoints those 2Dkeypoints with match values that satisfy a particular threshold.Additionally or alternatively, the system 900 may reject as matchesthose 2D keypoints that have a first-second best match ratio that isless than a threshold (e.g., less than 1).

Those 2D keypoints that are identified as matches may be selected forinclusion in the subset of keypoints. Additionally or alternatively,those 2D keypoints that are not identified as matches may be excludedfrom the subset of 2D keypoints in some embodiments. In these or otherembodiments, those matching 2D keypoints that match in multiple of thekeyframes may be selected for inclusion in the subset of 2D keypoints.Additionally or alternatively, the 2D keypoints that match in athreshold number of keyframes may be selected for inclusion in thesubset of 2D keypoints.

In these or other embodiments, the system 900 may perform matching bothways between different keyframes. For example, the system 900 mayperform matching from a first keyframe to a second keyframe and thenback from the second keyframe to the first keyframe. Those 2D keypointsthat are identified as matches in both matching directions may beselected for inclusion in the subset of 2D keypoints. Additionally oralternatively, those 2D keypoints that are not identified as matches inboth matching directions may be excluded from the subset of 2D keypointsin some embodiments.

In some embodiments, the system 900 may determine correspondencesbetween the 2D keypoints and LIDAR points of the LIDAR scans associatedwith the keyframes. In these or other embodiments, the system 900 may beconfigured to determine the subset of 2D keypoints based on thedetermined correspondences.

For example, the system 900 may be configured to identify LIDAR pointsin the LIDAR scans that correspond to the 2D keypoints. LIDAR points inthe LIDAR scans that correspond to the same features as the 2D keypointsmay be those LIDAR points that correspond to the 2D keypoints. In someembodiments, 2D keypoints that are identifiable as corresponding toLIDAR points may be included in the subset. Additionally oralternatively, LIDAR points that correspond to matching 2D keypointsthat are identified as matching may also be compared. In response to thecorresponding LIDAR points being the same or being within a particulardistance of each other, the particular matching 2D keypoints may beselected for inclusion in the subset of 2D keypoints. In these or otherembodiments, in response to the corresponding LIDAR points not being thesame or not being within the particular distance of each other, theparticular matching 2D keypoints may be rejected for inclusion in thesubset of keypoints.

In some embodiments, the selected 2D keypoints may be used to generaterespective 3D keypoints. For example, matching 2D keypoints thatcorrespond to a same particular feature may be used to generate acorresponding 3D keypoint that corresponds to the particular feature.The 2D keypoints may comprise a 2D representation of the particularfeature and the corresponding 3D keypoint may comprise a 3Drepresentation of the particular feature. The generation of the 3Dkeypoints based on the 2D keypoints may be performed using any suitabletechnique, such as a triangulation technique.

In some embodiments, the system 900 may determine correspondencesbetween the 3D keypoints and LIDAR points of the LIDAR scans associatedwith the keyframes. In these or other embodiments, the system 900 may beconfigured to filter the 3D keypoints based on the determinedcorrespondences. For example, the system 900 may be configured toidentify LIDAR points in the LIDAR scans that correspond to the 3Dkeypoints. LIDAR points in the LIDAR scans that correspond to the samefeatures as the 3D keypoints may be those LIDAR points that correspondto the 3D keypoints. In some embodiments, 3D keypoints that areidentifiable as corresponding to LIDAR points may be kept as part of thefiltering.

In these or other embodiments, the system 900 may filter the 3Dkeypoints based on one or more semantics of the 3D keypoints that mayindicate characteristics of the corresponding features or objects of the3D keypoints. For example, some characteristics of the features of the3D keypoints may be more beneficial than others for certainapplications, such as localization based on the 3D keypoints. Forinstance, relatively smooth and uniform surfaces such as of a pole maybe more useful for image based localization than a bush. Therefore, insome embodiments, the 3D keypoints may be filtered according toidentified semantics and the corresponding features thereof. In thepresent disclosure, reference to characteristics of 3D keypoints mayrefer to the characteristics of the features or objects that correspondto the 3D keypoints.

In these or other embodiments, the characteristics of the 3D keypointsmay be determined based on the LIDAR points that correspond to the 3Dkeypoints. For example, the LIDAR points that correspond to a particular3D keypoint and surrounding LIDAR points within a particular radius ofthe corresponding LIDAR points may be evaluated to determine thecharacteristics of the objects or features that correspond to such LIDARpoints. LIDAR points that are relatively consistent with each other mayindicate smooth surfaces or similar characteristics that may bebeneficial. Accordingly, in some embodiments, 3D keypoints thatcorrespond to LIDAR points and surrounding LIDAR points with determinedsmooth characteristics may be kept. Conversely, neighboring LIDAR pointsthat have inconsistencies with respect to each other may indicaterelatively rough or fuzzy surfaces, which may not be as useful.Accordingly, in some embodiments, 3D keypoints that correspond to LIDARpoints and surrounding LIDAR points with determined rough or fuzzycharacteristics may be filtered.

As indicated above, in the illustrated example of FIG. 9B, the keypoints970 may be 3D keypoints. The system 900 may be configured to annotatethe HD map data with keypoint data that indicates the locations of thekeypoints 970 in the HD map 912. For example, based on the locations inthe HD map 912 of the map points that correspond to the 3D keypoints(e.g., as identified from the LIDAR scans) the locations of thekeypoints 970 within the HD map 912 may be identified. The map pointsthat correspond to the keypoints 970 may be annotated as correspondingto respective keypoints 970 such that the locations of the keypoints 970may be indicated in the HD map data. In some embodiments, the locationsof one or more keypoints 970 may be determined based on matching 2Dkeypoints that each correspond to the same features and used to generatethe same 3D keypoint. Additionally or alternatively, multiple of thesame 3D keypoints may be generated with respect to different matching 2Dkeypoints from different keyframes and the locations of one or more ofthe keypoints 970 may be based on the locations of the differentmatching 3D keypoints.

For example, the system 900 may project the matching keypoints (2Dand/or 3D) onto the HD map 912 by identifying the corresponding mappoints of the matching keypoints. The corresponding map points maydiffer for the different matching keypoints based on differentperspectives associated with the keyframes associated with the matchingkeypoints. In some embodiments, the system 900 may determine which ofthe map points (and corresponding map locations) to associate with thekeypoint of the matching keypoints based on reducing or minimizing areprojection error that relates to discrepancies between which mappoints correspond to the common features of the matching keypoints.

In these or other embodiments, the keypoint data may indicate theidentified features of the keypoints. Additionally or alternatively, thekeypoint data may indicate and/or include the keyframes that include therespective keypoints. In these or other embodiments, the keypoint datamay include the 3D and/or 2D representation of the features or objectsthat correspond to the respective keypoints 970. In some embodiments,the system 900 may store the six-dimensional alignment, location, and/ororientation of the keypoints 970 in the HD map data as part ofindicating the location of the keypoints 970 in the HD map 912.

Image-Based Localization

As indicated above, the annotation of the HD map data with the keypointdata may improve image-based localization. An example of which isdescribed below with respect to the vehicle 950 travelling in the region901 represented by the HD map 912. In some embodiments, the system 900(e.g., via the vehicle computing system 920) may determine poses of thevehicle 950 by comparing data obtained from LIDARs, HD maps, GPSs,camera images, videos, etc. In some embodiments, a pose may include arepresentation of the alignment, location, and/or orientation of thevehicle 950 based on sensor data and/or data obtained from LIDARs, HDmaps, GPSs, camera images, videos, etc. In some embodiments, the posemay be used for alignment, location, and/or orientation purposes. Insome embodiments, the pose may be used to identify one or more sets ofkeypoints 970 that may individually and/or collectively represent thealignment, location, and/or orientation of the vehicle 950 relative tosensor data and/or data obtained from LIDARs, HD maps, GPSs, cameraimages, videos, etc. In some embodiments, the pose may include asix-dimensional alignment, location, and/or orientation (e.g., x, y, z,roll, pitch, and yaw) of the vehicle 950 based on sensor data and/ordata obtained from LIDARs, HD maps, GPSs, camera images, videos, etc.For example, the system 900 may obtain sensor data from a LIDAR sensorfixed to the vehicle 950 from the region 901 and HD map data for thesame or a similar region.

In some embodiments, the vehicle 950 may have one or more camerasmounted to a portion of the vehicle 950. In some embodiments, as thevehicle 950 is driving along a given path, one or more cameras of thevehicle 950 may capture images and the system 900 may compare the cameraimages against the keypoints 970 within a threshold distance of a sensorpose 972 of the vehicle 950. In some embodiments, the system 900 mayfind keypoint correspondences between the camera images and thekeypoints 970 and may use the keypoint correspondences to determine acamera pose 976 of the vehicle 950.

For example, in some embodiments, a camera of the vehicle 950 maycapture images and the system 900 (e.g., via the vehicle computingsystem 920) may match points of interest of the camera images withpoints of interest of the keypoints 970 to find keypoint correspondencesfor determining poses of the vehicle 950. In some embodiments, thesystem 900 may also perform image-based localization by recognizing thekeypoints 970 across images by matching images. For example, the system900 may determine the keypoints 970 from a first image match with orcorrespond to one or more keypoints from a second image. In someembodiments, the system 900 may determine a camera pose of a vehicle 950based on the matching of camera images with previously identifiedkeypoints 970. For example, the system 900 may determine a camera pose976 of the vehicle 950 based on keypoints 970 that may be determinedfrom a camera image that may match with or correspond to one or morepreviously identified keypoints 970 from a second camera image or fromanother source.

The keypoints 970 may each have associated therewith correspondingkeypoint data, that may be included in the HD map data of the HD map912. The keypoint data may represent a respective feature of the region901 at a corresponding location of the respective keypoint 970. Forexample, first keypoint data may represent a first feature of the region901 at a first location of the keypoint 970 a, second keypoint data mayrepresent a second feature of the region 901 at a second location of thekeypoint 970 b, etc. In some embodiments, the keypoint data may includeimages of the respective keypoints 970 and/or representations offeatures of the respective keypoints 970 that may indicate features thatmay be identified from images captured by cameras. Further, thekeypoints 970 and corresponding features may be selected based ongeneral uniqueness of the features and the keypoints and relativeuniqueness with respect to each and surrounding features such that thekeypoints 970 are relatively easier to identify and be distinguishedfrom other elements of the region 901.

In some embodiments, the computing system 920 may compare sensor datafrom the vehicle sensors 905 and HD map data from the HD map system 910to determine a sensor pose 972 illustrated in FIG. 9B. The sensor pose972 may represent the alignment, location, and/or orientation of thesensor and/or vehicle 950 relative to the HD map 912 of the region 901with a first degree of accuracy.

In some embodiments, the system 900 may use sensor pose 972 as aninitial estimate of the location of the vehicle 950 in the region 901.In some embodiments, the sensor pose 972 may have a particular degree ofaccuracy. The degree of accuracy may be a measurement of the accuracy ofa representation of the alignment, location, and/or orientation of thevehicle sensors 905 and/or vehicle 950 in the region 901 as comparedwith the actual alignment, location, and/or orientation of the vehiclesensors 905 and/or vehicle 950 in the region 901. For example, theparticular degree of accuracy of the sensor pose 972 may be within 1-3meters or better.

In some embodiments, the particular degree of accuracy of the sensorpose 972 may be limited by the inaccuracies and/or errors in the dataused to determine the sensor pose 972. For example, the system 900 mayuse vehicle sensor data from vehicle sensors 905 that may be obtainedfrom LIDAR scans from one or more LIDARs or GPS data that may beobtained from one or more GPSs and HD map data to determine the sensorpose 972. The vehicle sensor data and HD map data may be subject to theinaccuracies and/or errors introduced by one or more of the LIDAR, GPS,and HD map 912. Additionally or alternatively, due to the rates at whichthe vehicle sensor data may be obtained and depending on the velocity ofthe vehicle 950, a first sensor pose 972 determined with respect to afirst sensor data that is read at a time “t1” may be inaccurate withrespect to an actual location of the vehicle at a time “t2” that isafter time “t1” but before a second sensor reading time “t3” of secondsensor data that may be used to determine a second sensor pose 972.

In some embodiments, the system 900 may use the sensor poses 972 as astarting place to determine a more precise estimate of the alignment,location, and/or orientation of the vehicle sensors 905 and/or vehicle950 in the region 901.

In some embodiments, the system 900 may use the sensor poses 972 as astarting place to determine a more precise estimate of the alignment,location, and/or orientation of the vehicle sensors 905 and/or vehicle950 in the region 901. In some embodiments, the system 900 may use thesensor poses 972 and one or more sets of keypoints 970 to determine oneor more camera poses 976 to obtain the more precise estimate. In someembodiments, the camera poses 976 may have a particular degree ofaccuracy. In some embodiments, the camera poses 976 may have a higherdegree of accuracy than the sensor poses 972. For example, theparticular degree of accuracy of the sensor poses 972 may be within 30cm or better. In some embodiments, the camera pose 976 may be used toalign, locate, and/or orient a camera and/or vehicle or may be used toassist in the navigation of a vehicle.

For example, the vehicle 950 may have moved in a direction indicated byarrow 980 since the capturing of the sensor data (e.g., LIDAR data) usedto determine the sensor pose 972. The speed of the vehicle 950 and theamount of time may thus affect the first degree of accuracy of thesensor pose 972. For example, as illustrated in FIG. 9B, the sensor pose972 may be toward the back of the vehicle 950 but may be meant toindicate the pose of the vehicle 950 toward the front of the vehicle950.

As another example, the sensor pose 972 may be obtained based on GPSdata that may have a margin of error of a few meters. In the illustratedexample of FIG. 9B, the sensor pose 972 may be toward the back of thevehicle 950 instead of the toward the front of the vehicle 950 eventhough the sensor pose 972 is meant to be toward the front of thevehicle 950 due to the GPS inaccuracy. The first degree of accuracy inthis example may correspond to the degree of accuracy of the GPS data.

In some embodiments, the system 900 may perform the followingoperations. In some embodiments, the system 900 may associate points orsets of points in a point cloud of the HD map 910 (e.g., OMap) withfeatures and may identify them as keypoints 970 as described above. Insome embodiments, the system 900 may be configured to translate, rotate,and scale the point cloud of the HD map 910. In some embodiments, asalso indicated above, the system 900 may store locations of thekeypoints 970 with the HD map data of the HD map 912. In someembodiments, the system 900 may use these stored locations to identifykeypoints 970 within a specified distance of a location, for example, byperforming a radius search, for example, using a radius 974.Accordingly, in some embodiments, the system 900 may determine anapproximate location of the vehicle 950 by matching sensor scan datawith the point cloud of the HD map 910 (e.g., using an ICP (iterativeclosest point) technique) and may identify keypoints 970 that may belikely to be visible from the location of the vehicle 950. In someembodiments, the system 900 may search for the identified keypoints 970in the camera images that may be captured by the vehicle 950. In someembodiments, the system 900 may be able to perform a sensor pose 972determination at a lower frequency compared to camera pose 976determination because capturing a sensor scan may take longer thancapturing a camera image. Accordingly, in some embodiments, for eachsensor pose determination, the system 900 may perform a plurality ofcamera pose 976 determinations until the computing system 920 mayperform a subsequent sensor pose 972 determination.

In some embodiments, the computing system 920 may identify a set ofkeypoints 970 that may be within a threshold distance of the sensor pose972. For example, the computing system 920 may identify a set ofkeypoints, including keypoints 970 b, 970 c, 970 d, and 970 f, which maybe within the radius 974 of the sensor pose 972.

In some embodiments, the set of keypoints that are within the thresholddistance of the sensor pose 972 may be used to determine a camera pose976 of the vehicle 950. For example, the computing system 920 may obtainone or more camera images of the region 901. For example, the computingsystem 920 may obtain one or more camera images of the region 901 thatare captured by one or more cameras the vehicle sensors 905. Thecomputing system 920 may compare the one or more camera images againstthe keypoint data of the set of keypoints, for example, keypoint dataassociated with keypoints 970 b, 970 c, 970 d, and 970 f, to identifyone or more matching keypoints that are included in the one or morecamera images. The matching keypoints may be those keypoints 970 of theset of the keypoints 970 b, 970 c, 970 d, and 970 f that may be depictedin the camera images.

The comparing of the keypoint data against the camera images may includecomparing the camera images against images included in the keypointdata. Additionally or alternatively, the comparing of the keypoint dataagainst the camera images may include performing image processing on thecamera images to identify particular features of objects included in thecamera images and comparing the identified features to those included inthe keypoint data. The features included in the keypoint data may beidentified by performing image processing on images included in thekeypoint data in some embodiments. Additionally or alternatively, thecomputing system 920 may be configured to focus mainly on or prioritizethe keypoint data of the set of keypoints 970 b, 970 c, 970 d, and 970 fin comparing the captured images against other HD map data of the region901. In these or other embodiments, the computing system 920 may onlycompare the images against the keypoint data of the set of keypoints 970b, 970 c, 970 d, and 970 f.

The computing system 920 may determine a camera pose 976 of the vehicle950 based on the one or more matching keypoints 970. For example, thecomputing system 920 may determine a camera pose 976 of the vehicle 950based on the one or more matching keypoints 970 b, 970 c, 970 d, and 970f. The camera pose 976 may indicate a location of the vehicle 950 withinthe HD map 912. In some embodiments, the camera pose 976 may indicate alocation of the vehicle 950 within the HD map 912, with a second degreeof accuracy which may be higher than the first degree of accuracy of thesensor pose 972. The uniqueness of the keypoints 970 and the focusing onthe keypoint data in determining matching keypoints and the camera pose976 may allow for a faster and more accurate determination of the camerapose 976 by directing the image comparing toward elements that may beeasier to identify and more accurately identified.

In some embodiments, the vehicle computing system 920 may use thekeypoint data for the keypoints 970 b, 970 c, 970 d, and 970 f, sensorpose 972, camera pose 976, etc. to the vehicle computing system 920 toassist with the navigation of the vehicle 950.

In some embodiments, the system 900 may receive sensor scans from one ormore vehicle sensors 905 as the vehicle 950 navigates along a route. Insome embodiments, the system 900 may determine the sensor pose 972 basedon the sensor data. In some embodiments, the system 900 may use thesensor pose 972 to identify keypoints 970 near the sensor pose 972, forexample, by identifying keypoints 970 within a threshold distance of thesensor pose 972 as determined by performing a search within apredetermined radius, for example, within the radius 974. In someembodiments, the threshold distance may be based on the Euclideandistance between the sensor pose 972 and the identified keypoints 970.The system 900 may receive camera images and may perform matches betweenthe identified set of keypoints 970 and camera image data to determinecorrespondences between camera images and keypoints 970. In someembodiments, the sensor-based localization based on the sensor pose 972may help narrow the search for the image-based localization to a smallset of keypoints 970 that may be local to the position of the vehicle950. Accordingly, in some embodiments, the image-based localizationtechniques disclosed herein may improve performance of the system 900 ormay improve the precision, accuracy, or speed of the vehiclelocalization because the search for keypoints 970 may performed using asmall set of keypoints 970 that may be local to the given region 901where the vehicle 950 may be determined to be present based on sensorlocalization. In some embodiments, the image-based localization may alsoassist to fully constrain or more fully constrain the alignment,location, and/or orientation of the camera and/or vehicle 950, to reduceuncertainty, mismatches, and/or errors that may be present in the sensorlocalization data.

In some embodiments, the system 900 may determine and storetransformations for mapping from a sensor pose 972 to a camera pose 976based on the known position of the camera with respect to the sensor.Accordingly, in some embodiments, given a LIDAR-based pose, the system900 can determine a corresponding camera pose 976.

In some embodiments, the system 900 may use scale-invariant featuretransform (SIFT) to identify keypoints 970 and keyframes. Details of theSIFT algorithm may be disclosed in the U.S. Pat. No. 6,711,293 filed onMar. 6, 2000 which is incorporated by reference herein. The SIFTtechniques may transform an image into a collection of feature vectors,each of which may be invariant to image translation, scaling, androtation, and may be partially invariant to illumination changes androbust to local geometric distortion.

In some embodiments, the system 900 may perform RANSAC (random sampleconsensus) based on triangulation for matching keypoints of a new imagewith previously stored keypoints 970.

In some embodiments, the system 900 may associate points with semanticsand may use the semantics to make keypoint matching more reliable.Accordingly, in some embodiments, the system may determine semantics ofthe objects in camera images and may compare or match them withsemantics of objects in the HD map to perform correspondences.

In some embodiments, the system 900 may be able to use the sensor poses972 to shorten the time and/or resources required to determine a moreprecise estimate of the alignment, location, and/or orientation of thesensor and/or vehicle 950 in a given region 901. For example, the system900 may use one or more sensor poses 972 to narrow the region from whichone or more camera images are obtained. The narrowing of the region mayreduce the number of mismatches that may occur between one or more setsof keypoints 970 within a threshold distance of the one or more sensorposes 972 and the camera pose 976 of the vehicle 950. The reduction ofthe number of mismatches may result in error reduction in thelocalization of the vehicle 950 and less overall drain on the system900. In some instances, an additional, more precise, and/or modifiedestimate of the alignment, location, and/or orientation may be utilizedto more accurately navigate the vehicle 950 through the region 901.

FIG. 10 is a flowchart of an example method 1000 of image-basedlocalization for vehicles, according to at least one embodimentdescribed in the present disclosure. The method 1000 may be performed byany suitable system, apparatus, or device. For example, one or moreelements of the HD map system 100 of FIG. 1 may be configured to performone or more of the operations of the method 1000. Additionally oralternatively, the vehicle computing system 120 may be configured toperform one or more of the operations associated with the method 1000.Although illustrated with discrete blocks, the steps and operationsassociated with one or more of the blocks of the method 1000 may bedivided into additional blocks, combined into fewer blocks, oreliminated, depending on the particular implementation.

At block 1002, vehicle sensor data related to a region through which avehicle may be driving may be received. The vehicle sensor data may bereceived via the vehicle computing system 120 and may include dataobtained from LIDAR scans from one or more LIDARs, GPS data obtain fromone or more GPS s, and/or HD map data.

At block 1004, HD map data related to an HD map of the region may beaccessed. The HD map data may comprise map sensor data and may identifylocations within the HD map of a plurality of keypoints of the region inwhich each respective keypoint has associated corresponding keypointdata that may represent a respective feature of the region at acorresponding location of the respective keypoint. The vehicle computingsystem 120 may receive the HD map data from the online map system.

At block 1006, a sensor pose of the vehicle may be determined bycomparing the vehicle sensor data with the map sensor data. The sensorpose may indicate a location of the vehicle within the HD map within afirst degree of accuracy.

At block 1008, a set of keypoints from the plurality of keypoints of theregion based on each keypoint of the set of keypoints being within athreshold distance of the sensor pose of the vehicle in the HD map maybe identified. The identification of the set of keypoints from theplurality of keypoints of the region may be indicated by the HD mapdata. The identifying the set of keypoints from the plurality ofkeypoints may comprise identifying one or more objects in the HD mapdata and extracting one or more points of interest from one or moreobjects. Each keypoint of the set of keypoints may correspond to one ormore points of interest within the HD map data.

At block 1010, one or more camera images of the region captured by oneor more cameras mounted on the vehicle may be obtained.

At block 1012, the one or more camera images may be compared againstkeypoint data of the set of keypoints to identify one or more matchingkeypoints that are included in the one or more camera images. Thecomparing the one or more camera images against keypoint data of the setof keypoints may comprise identifying one or more objects in the one ormore camera images and extracting one or more points of interest fromthe one or more objects. The one or more matching keypoints maycorrespond to one or more points of interest within the one or morecamera images.

At block 1014, a camera pose of the vehicle based on the one or morematching keypoints may be determined. The camera pose may indicate thelocation of the vehicle within the HD map within a second degree ofaccuracy that may be higher than the first degree of accuracy.

Modifications, additions, or omissions may be made to the method 1000without departing from the scope of the present disclosure. For example,the operations of method 1000 may be implemented in differing order.Additionally or alternatively, two or more operations may be performedat the same time. Furthermore, the outlined operations and actions areonly provided as examples, and some of the operations and actions may beoptional, combined into fewer operations and actions, or expanded intoadditional operations and actions without detracting from the essence ofthe disclosed embodiments.

FIG. 11 is a flowchart of an example method 1100 of image-based keypointgeneration, according to at least one embodiment described in thepresent disclosure. The method 1100 may be performed by any suitablesystem, apparatus, or device. For example, one or more elements of theHD map system 100 of FIG. 1 may be configured to perform one or more ofthe operations of the method 1100. Additionally or alternatively, thevehicle computing system 120 may be configured to perform one or more ofthe operations associated with the method 1100. Although illustratedwith discrete blocks, the steps and operations associated with one ormore of the blocks of the method 1100 may be divided into additionalblocks, combined into fewer blocks, or eliminated, depending on theparticular implementation.

At block 1102, a plurality of scans may be obtained for a region. Insome embodiments, the scans may include LIDAR scans of the region. Insome embodiments, the LIDAR scans may be obtained such as describedabove with respect to FIGS. 9A and 9B. For example, the LIDAR scans maybe obtained by one or more vehicles travelling through the region.Additionally or alternatively, the LIDAR scans may be included in an HDmap of the region.

At block 1104, a plurality of LIDAR poses may be identified. In someembodiments, each LIDAR pose of the plurality of LIDAR poses maycorrespond to a respective LIDAR scan of the plurality of LIDAR scans.In some embodiments, the LIDAR poses may be identified such as describedabove with respect to FIGS. 9A and 9B. For example, identifying theLIDAR poses may include obtaining the LIDAR poses from map data of theHD map of the region. In these or other embodiments, identifying theLIDAR poses may include determining one or more of the LIDAR poses basedon localization performed with respect to the corresponding the LIDARscans of the one or more LIDAR poses. Additionally or alternatively,identifying the plurality of LIDAR poses includes determining theplurality of LIDAR poses with respect to relative locations of eachother.

At block 1106, a plurality of images of the region that are capturedduring capturing of the plurality of LIDAR scans may be identified askeyframes. In some embodiments, the keyframes may be identified such asdescribed above with respect to FIGS. 9A and 9B. For example, in someembodiments, identifying the keyframes may include selecting keyframesthat correspond to a particular distance between camera poses thatsatisfy a threshold distance difference between each other.

At block 1108, a plurality of camera poses may be identified based onthe plurality of LIDAR poses. In some embodiments, each respectivecamera pose of the plurality of camera poses may correspond to arespective keyframe of the plurality of keyframes. In some embodiments,the camera poses may be identified such as described above with respectto FIGS. 9A and 9B.

At block 1110, a plurality of two-dimensional (2D) keypoints may beidentified. Identifying the 2D keypoints may include identifying one ormore respective 2D keypoints in a corresponding keyframe of each of thekeyframes. In some embodiments, each respective 2D keypoint correspondsto a respective feature of the region as included in the correspondingkeyframe. In some embodiments, the 2D keypoints may be identified suchas described above with respect to FIGS. 9A and 9B.

For example, identifying the 2D keypoints may include identifying, asthe 2D keypoints, matching keypoints that each correspond to multiple ofthe keyframes. In these or other embodiments, identifying the 2Dkeypoints may include identifying one or more common features thatcorrespond to multiple keyframes and identifying the one or more commonfeatures as one or more of the 2D keypoints. Additionally oralternatively, identifying the 2D keypoints may include identifying asthe 2D keypoints, matching keypoints that each correspond to a thresholdnumber of the keyframes.

At block 1112, one or more three-dimensional (3D) keypoints may begenerated. The 3D keypoints may be generated based on the plurality of2D keypoints and the respective camera poses of the respective keyframesof the plurality of keyframes, such as described above.

Modifications, additions, or omissions may be made to the method 1100without departing from the scope of the present disclosure. For example,the operations of method 1100 may be implemented in differing order.Additionally or alternatively, two or more operations may be performedat the same time. Furthermore, the outlined operations and actions areonly provided as examples, and some of the operations and actions may beoptional, combined into fewer operations and actions, or expanded intoadditional operations and actions without detracting from the essence ofthe disclosed embodiments.

For example, in some embodiments, the method 1100 may includedetermining correspondences between LIDAR points of the plurality ofLIDAR scans and the one or more 3D keypoints and filtering the one ormore 3D keypoints based on the determined correspondences. In these orother embodiments, the filtering may include removing 3D keypoints thatdo not correspond to LIDAR points of the plurality of LIDAR scans,keeping 3D keypoints that correspond to one or more respective LIDARpoints of the LIDAR scans, keeping 3D keypoints that correspond to athreshold number of respective LIDAR points of the LIDAR scans, or anysuitable combination thereof. Additionally or alternatively, thefiltering may be based on determined characteristics of the one or more3D keypoints. The determined characteristics may be based on one or moreof: determined semantics of the 3D keypoints; or determinedcharacteristics of LIDAR points that correspond to the 3D keypoints insome embodiments.

Additionally or alternatively, the method 1100 may include annotatingmap data of a high definition (HD) map of the region with keypoint datathat corresponds to the one or more 3D keypoints. For example, themethod 1100 may include identifying a plurality of locations within theHD map in which each respective location of the plurality of locationscorresponds to a respective 3D keypoint and annotating map data of theHD map with keypoint data that indicates the respective locations of therespective 3D keypoints.

Computer System Architecture

FIG. 12 is a block diagram illustrating various components of an examplecomputer system that may read instructions from a machine-readablemedium and may execute them in a processor (or controller).Specifically, FIG. 12 illustrates a diagrammatic representation of amachine in the example form of a computer system 1200 within whichinstructions 1224 (e.g., software) for causing the machine to performany one or more of the methodologies discussed herein may be executed.In some embodiments, the machine may operate as a standalone device ormay be connected (e.g., networked) to other machines. In someembodiments, the machine may operate, in a networked development, in thecapacity of a server machine or a client machine in a server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment.

In some embodiments, the machine may be a server computer, a clientcomputer, a personal computer (PC), a tablet PC, a set-top box (STB), apersonal digital assistant (PDA), a cellular telephone, a smartphone, aweb appliance, a network router, switch or bridge, or any machinecapable of executing instructions 1224 (sequential or otherwise) thatmay specify actions to be taken by the machine. Further, while only asingle machine is illustrated, the term “machine” may also be taken toinclude any collection of machines that individually or jointly executeinstructions 1224 to perform any one or more of the methodologiesdiscussed herein.

In some embodiments, the example computer system 1200 may be part of ormay be any applicable system described in the present disclosure. Forexample, the online HD map system 120 and/or the vehicle computingsystem 120 described above may comprise the computer system 1200 or oneor more portions of the computer system 1200. Further, in someembodiments, different implementations of the computer system 1200 mayinclude more or fewer components than those described herein. Forexample, a particular computer system 1200 may not include one or moreof the elements described herein and/or may include one or more elementsthat are not explicitly discussed.

In some embodiments, the example computer system 1200 may include aprocessor 1202 (e.g., a central processing unit (CPU), a graphicsprocessing unit (GPU), a digital signal processor (DSP), one or moreapplication specific integrated circuits (ASICs), one or moreradio-frequency integrated circuits (RFICs), or any combination ofthese), a main memory 1204, and a static memory 1206, which may beconfigured to communicate with each other via a bus 1208. In someembodiments, the computer system 1200 may further include graphicsdisplay unit 1210 (e.g., a plasma display panel (PDP), a liquid crystaldisplay (LCD), a projector, or a cathode ray tube (CRT)). In someembodiments, the computer system 1200 may also include alphanumericinput device 1212 (e.g., a keyboard), a cursor control device 1214(e.g., a mouse, a trackball, a joystick, a motion sensor, or otherpointing instrument), a storage unit 1216, a signal generation device1218 (e.g., a speaker), and a network interface device 1220, which maybe configured to communicate via the bus 1208.

In some embodiments, the storage unit 1216 may include amachine-readable medium 1222 on which is stored instructions 1224 (e.g.,software) embodying any one or more of the methodologies or functionsdescribed herein. In some embodiments, the instructions 1224 (e.g.,software) may also reside, completely or at least partially, within themain memory 1204 or within the processor 1202 (e.g., within aprocessor's cache memory) during execution thereof by the computersystem 1200, the main memory 1204 and the processor 1202 alsoconstituting machine-readable media. The instructions 1224 (e.g.,software) may be transmitted or received over a network 1226 via thenetwork interface device 1220.

While machine-readable medium 1222 is shown in an example embodiment tobe a single medium, the term “machine-readable medium” may be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storeinstructions (e.g., instructions 1224). The term “machine-readablemedium” may also be taken to include any medium that is capable ofstoring instructions (e.g., instructions 1224) for execution by themachine and that cause the machine to perform any one or more of themethodologies disclosed herein. The term “machine-readable medium” mayinclude, but not be limited to, data repositories in the form ofsolid-state memories, optical media, and magnetic media.

ADDITIONAL CONFIGURATION CONSIDERATIONS

The foregoing description has been presented for the purpose ofillustration; it is not intended to be exhaustive or to be limiting inany way. Persons skilled in the relevant art may appreciate that manymodifications and variations may be possible in light of the abovedisclosure.

For example, although the techniques described herein may be applied toautonomous vehicles, the techniques can also be applied to otherapplications, for example, for displaying HD maps for vehicles withdrivers, for displaying HD maps on displays of client devices such asmobile phones, laptops, tablets, or any computing device with a displayscreen. Techniques displayed herein may also be applied for displayingmaps for purposes of computer simulation, for example, in computergames, etc.

Some portions of this description may describe the embodiments in termsof algorithms and symbolic representations of operations on information.These algorithmic descriptions and representations may be commonly usedby those skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, may beunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, etc. Furthermore, it may be provenconvenient at times, to refer to these arrangements of operations asmodules, without loss of generality. The described operations and theirassociated modules may be embodied in software, firmware, hardware, orany combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In some embodiments, asoftware module may be implemented with a computer program productcomprising a computer-readable medium containing computer program code,which may be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for thedisclosed purposes, and/or it may comprise a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in atangible computer readable storage medium or any type of media suitablefor storing electronic instructions and coupled to a computer systembus. Furthermore, any computing systems referred to in the specificationmay include a single processor or may be architectures employingmultiple processor designs for increased computing capability.

Embodiments of the invention may also relate to a computer data signalembodied in a carrier wave, where the computer data signal may includeany embodiment of a computer program product or other data combinationdescribed herein. The computer data signal may be a product that may bepresented in a tangible medium or carrier wave and modulated orotherwise encoded in the carrier wave, which may be tangible, andtransmitted according to any suitable transmission method.

In addition, the language used in the specification may have beenprincipally selected for readability and instructional purposes and maynot have been selected to delineate or circumscribe the inventivesubject matter. It may therefore be intended that the scope of theinvention may not be limited by this detailed description but rather byany claims that issue on an application based hereon.

As used herein, the terms “module” or “component” may refer to specifichardware implementations configured to perform the operations of themodule or component and/or software objects or software routines thatmay be stored on and/or executed by general-purpose hardware (e.g.,computer-readable media, processing devices, etc.) of the computingsystem. In some embodiments, the different components, modules, engines,and services described herein may be implemented as objects or processesthat execute on the computing system (e.g., as separate threads). Whilesome of the system and methods described herein are generally describedas being implemented in software (e.g., stored on and/or executed bygeneral purpose hardware), specific hardware implementations or acombination of software and specific hardware implementations are alsopossible and contemplated. In this description, a “computing entity” maybe any computing system as previously defined herein, or any module orcombination of modulates running on a computing system.

Terms used in the present disclosure and especially in the appendedclaims (e.g., bodies of the appended claims) are generally intended as“open” terms (e.g., the term “including” should be interpreted as“including, but not limited to,” the term “having” should be interpretedas “having at least,” the term “includes” should be interpreted as“includes, but is not limited to,” etc.).

Additionally, if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis explicitly recited, those skilled in the art will recognize that suchrecitation should be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, means at least two recitations, or two or more recitations).Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” or “one or more of A, B, and C, etc.”,or “at least one of A, B, or C, etc.” or “one or more of A, B, or C,etc.” is used, in general such a construction is intended to include Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, or A, B, and C together, etc. Additionally, the use of theterm “and/or” is intended to be construed in this manner.

Further, any disjunctive word or phrase presenting two or morealternative terms, whether in the description, claims, or drawings,should be understood to contemplate the possibilities of including oneof the terms, either of the terms, or both terms. For example, thephrase “A or B” should be understood to include the possibilities of “A”or “B” or “A and B” even if the term “and/or” is used elsewhere.

All examples and conditional language recited in the present disclosureare intended for pedagogical objects to aid the reader in understandingthe present disclosure and the concepts contributed by the inventor tofurthering the art and are to be construed as being without limitationto such specifically recited examples and conditions. Althoughembodiments of the present disclosure have been described in detail,various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the present disclosure.

What is claimed is:
 1. A method, comprising: obtaining a plurality oflight detection and ranging (LIDAR) scans of a region; identifying aplurality of LIDAR poses in which each LIDAR pose of the plurality ofLIDAR poses corresponds to a respective LIDAR scan of the plurality ofLIDAR scans; identifying, as a plurality of keyframes, a plurality ofimages of the region that are captured during capturing of the pluralityof LIDAR scans; determining, based on the plurality of LIDAR poses, aplurality of camera poses in which each respective camera pose of theplurality of camera poses corresponds to a respective keyframe of theplurality of keyframes; identifying a plurality of two-dimensional (2D)keypoints including identifying one or more respective 2D keypoints in acorresponding keyframe of each of the keyframes, wherein each respective2D keypoint corresponds to a respective feature of the region asincluded in the corresponding keyframe; and generating one or morethree-dimensional (3D) keypoints based on the plurality of 2D keypointsand the respective camera poses of the respective keyframes of theplurality of keyframes.
 2. The method of claim 1, wherein identifyingthe plurality of 2D keypoints includes identifying, as the plurality of2D keypoints, matching keypoints that each correspond to multiple of thekeyframes.
 3. The method of claim 1, further comprising: determiningcorrespondences between LIDAR points of the plurality of LIDAR scans andthe one or more 3D keypoints; and filtering the one or more 3D keypointsbased on the determined correspondences.
 4. The method of claim 3,wherein filtering the one or more 3D keypoints includes removing 3Dkeypoints that do not correspond to LIDAR points of the plurality ofLIDAR scans.
 5. The method of claim 3, wherein filtering the one or more3D keypoints is further based on determined characteristics of the oneor more 3D keypoints, wherein the determined characteristics are basedon one or more of: determined semantics of the 3D keypoints; ordetermined characteristics of LIDAR points that correspond to the 3Dkeypoints.
 6. The method of claim 3, wherein filtering the one or more3D keypoints includes keeping 3D keypoints that correspond to one ormore respective LIDAR points of the LIDAR scans.
 7. The method of claim3, wherein filtering the one or more 3D keypoints includes keeping 3Dkeypoints that correspond to a threshold number of respective LIDARpoints of the LIDAR scans.
 8. The method of claim 1, wherein theplurality of LIDAR scans are included in a high definition (HD) map ofthe region.
 9. The method of claim 8, further comprising: identifying aplurality of locations within the HD map in which each respectivelocation of the plurality of locations corresponds to a respective 3Dkeypoint; and annotating map data of the HD map with keypoint data thatindicates the respective locations of the respective 3D keypoints. 10.The method of claim 1, further comprising annotating map data of a highdefinition (HD) map of the region with keypoint data that corresponds tothe one or more 3D keypoints.
 11. The method of claim 1, whereinidentifying the plurality of keyframes includes selecting keyframes thatcorrespond to a particular distance between camera poses that satisfy athreshold distance difference between each other.
 12. The method ofclaim 1, wherein identifying the plurality of 2D keypoints includes:identifying one or more common features that correspond to multiplekeyframes; and identifying the one or more common features as one ormore of the plurality of 2D keypoints.
 13. The method of claim 1,wherein identifying the plurality of 2D keypoints includes identifyingas the plurality of 2D keypoints matching keypoints that each correspondto a threshold number of the keyframes.
 14. The method of claim 1,wherein identifying the plurality of LIDAR poses includes obtaining theplurality of LIDAR poses from map data of a high definition (HD) map ofthe region.
 15. The method of claim 1, wherein identifying the pluralityof LIDAR poses includes determining one or more of the plurality ofLIDAR poses based on localization performed with respect tocorresponding LIDAR scans.
 16. The method of claim 1, whereinidentifying the plurality of LIDAR poses includes determining theplurality of LIDAR poses with respect to relative locations of eachother.
 17. One or more non-transitory computer readable storage mediastoring instructions that, in response to being executed by one or moreprocessors, cause a system to perform operations, the operationscomprising: obtaining a plurality of light detection and ranging (LIDAR)scans of a region; identifying a plurality of LIDAR poses in which eachLIDAR pose of the plurality of LIDAR poses corresponds to a respectiveLIDAR scan of the plurality of LIDAR scans; identifying, as a pluralityof keyframes, a plurality of images of the region that are capturedduring capturing of the plurality of LIDAR scans; determining, based onthe plurality of LIDAR poses, a plurality of camera poses in which eachrespective camera pose of the plurality of camera poses corresponds to arespective keyframe of the plurality of keyframes; identifying aplurality of two-dimensional (2D) keypoints including identifying one ormore respective 2D keypoints in a corresponding keyframe of each of thekeyframes, wherein each respective 2D keypoint corresponds to arespective feature of the region as included in the correspondingkeyframe; and generating one or more three-dimensional (3D) keypointsbased on the plurality of 2D keypoints and the respective camera posesof the respective keyframes of the plurality of keyframes.
 18. The oneor more computer readable storage media of claim 17, wherein theoperations further comprise: determining correspondences between LIDARpoints of the plurality of LIDAR scans and the one or more 3D keypoints;and filtering the one or more 3D keypoints based on the determinedcorrespondences.
 19. A system comprising: one or more processors; andone or more non-transitory computer readable storage media storinginstructions that, in response to being executed by the one or moreprocessors, cause the system to perform operations, the operationscomprising: obtaining a plurality of light detection and ranging (LIDAR)scans of a region; identifying a plurality of LIDAR poses in which eachLIDAR pose of the plurality of LIDAR poses corresponds to a respectiveLIDAR scan of the plurality of LIDAR scans; identifying, as a pluralityof keyframes, a plurality of images of the region that are capturedduring capturing of the plurality of LIDAR scans; determining, based onthe plurality of LIDAR poses, a plurality of camera poses in which eachrespective camera pose of the plurality of camera poses corresponds to arespective keyframe of the plurality of keyframes; identifying aplurality of two-dimensional (2D) keypoints including identifying one ormore respective 2D keypoints in a corresponding keyframe of each of thekeyframes, wherein each respective 2D keypoint corresponds to arespective feature of the region as included in the correspondingkeyframe; and generating one or more three-dimensional (3D) keypointsbased on the plurality of 2D keypoints and the respective camera posesof the respective keyframes of the plurality of keyframes.
 20. Thesystem of claim 19, wherein the operations further comprise: determiningcorrespondences between LIDAR points of the plurality of LIDAR scans andthe one or more 3D keypoints; and filtering the one or more 3D keypointsbased on the determined correspondences.